A Lower Bound on the Estimator Variance 
for the Sparse Linear Model 
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Abstract — We study the performance of estimators of a sparse 
nonrandom vector based on an observation which is linearly 
transformed and corrupted by additive white Gaussian noise. 
Using the reproducing kernel Hilbert space framework, we derive 
a new lower bound on the estimator variance for a given differen- 
tiable bias function (including the unbiased case) and an almost 
arbitrary transformation matrix (including the underdetermined 
case considered in compressed sensing theory). For the special 
case of a sparse vector corrupted by white Gaussian noise — i.e., 
without a linear transformation — and unbiased estimation, our 
lower bound improves on previously proposed bounds. 

Index Terms — Sparsity, parameter estimation, sparse linear 
model, denoising, variance bound, reproducing kernel Hilbert 
space, RKHS. 

I. Introduction 

We study the problem of estimating a nonrandom parameter 
vector xgR" which is sparse, i.e., at most S of its entries are 
nonzero, where 1<S<N (typically S<^N). We thus have 



xeX s , with x s = { 



x'el 



dJVI 



< 



(1) 



where ||x|| denotes the number of nonzero entries of x. While 
the sparsity degree S is assumed to be known, the set of 
positions of the nonzero entries of x (denoted by supp(x)) is 
unknown. The estimation of x is based on the observed vector 



y G R M given by 



Hx 



(2) 



with a known system matrix H gR Mxff and white Gaussian 
noise n ~ 7V(0, er 2 I) with known variance a 2 >0. The matrix 
H is arbitrary except that it is assumed to satisfy the standard 
requirement 

spark(H) > S , (3) 

where spark(H) denotes the minimum number of linearly 
dependent columns of H [1]. The observation model (0 
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together with (HJ will be referred to as the sparse linear model 
(SLM). Note that we also allow M < N (this case is relevant 
to compressed sensing methods (TJ, 0); however, condition 
(O implies that M > S. The case of correlated Gaussian noise 
n with a known nonsingular correlation matrix can be reduced 
to the SLM by means of a noise whitening transformation. An 
important special case of the SLM is given by H = I (so that 
M = N), i.e., 

y = x + n , (4) 

where again x£ Xg and n ~ A/"(0, cr 2 I). This will be referred 
to as the sparse signal in noise model (SSNM). 

Lower bounds on the estimation variance for the SLM have 
been studied previously. In particular, the Cramer-Rao bound 
(CRB) for the SLM was derived in 0. For the SSNM ©, 
lower and upper bounds on the minimum variance of unbiased 
estimators were derived in flU. A problem with the lower 
bounds of J3] and (4 j is the fact that they exhibit a discontinuity 
when passing from the case ||x|| = 5 to the case ||x|| <5. 

In this paper, we use the mathematical framework of repro- 
ducing kernel Hilbert spaces (RKHS) [5 j-[7] to derive a novel 
lower variance bound for the SLM. The RKHS framework 
allows pleasing geometric interpretations of existing bounds, 
including the CRB, the Hammersley-Chapman-Robbins bound 
1181 , and the Barankin bound [9|. The bound we derive here 
holds for estimators with a given differentiable bias function. 
For the SSNM, in particular, we obtain a lower bound for 
unbiased estimators which is tighter than the bounds in ||4) 
and, moreover, everywhere continuous. As we will show, RKHS 
theory relates the bound for the SLM to that obtained for the 
linear model without a sparsity assumption. We note that the 
RKHS framework has been previously applied to estimation 
||6), but, to the best of our knowledge, not to the SLM. 

This paper is organized as follows. In Section |II] we review 
some fundamentals of parameter estimation. Relevant elements 
of RKHS theory are summarized in Section [III] In Section |IV] 
we use RKHS theory to derive a lower variance bound for 
the SLM. Section [V] considers the special case of unbiased 
estimation within the SSNM. Section |VI] presents a numerical 
comparison of the new bound with the variance of two estab- 
lished estimation schemes. 



II. Basic Concepts 

We first review some basic concepts of parameter estimation 
iflOl . Let x G X C be the nonrandom parameter vector 
to be estimated, y € M. M the observed vector, and /(y; x) the 
probability density function (pdf) of y, parameterized by x. For 
the SLM, X — X s as defined in (Q) and 

/(y; x ) = (27rt7 2 ) M/ 2 ex p(- ^ally-Hxin) • ( 5 ) 

A. Minimum-Variance Estimators 

The estimation error incurred by an estimator x(y) can 
be quantified by the mean squared error (MSE) e(x(-);x) = 
E x { ||x(y) — xlH}, where the notation E x {-} indicates that the 
expectation is taken with respect to the pdf /(y; x) parameter- 
ized by x. Note that e(x(-);x) depends on the true parameter 
value, x. The MSE can be decomposed as 

e(x(-);x) = ||b(*(.);x)||» + «(x(.);x), (6 ) 

with the estimator bias b(x(-);x) = E x {x(y)} — x and the 
estimator variance w(x(-); x) = E x { ||x(y) — E x {x(y)}||"}. A 
standard approach to defining an optimum estimator is to fix 

the bias, i.e., b(x(-);x) = c(x) for all x g X, and minimize 
the variance u(x(-);x) for all xG X under this bias constraint. 
However, in many cases, such a "uniformly optimum" estimator 
does not exist. It is then natural to consider "locally optimum" 
estimators that minimize i;(x(-);xo) only at a given parameter 
value x = xo G X. This approach is taken here. Note that it 
follows from © that once the bias is fixed, minimizing the 
variance is equivalent to minimizing the MSE e(x(-);xo). 

The bias constraint b(x(-);x) = c(x) can be equivalently 
written as the mean constraint 

E x {x(y)} = 7(x) , with -y(x) = c(x) + x . 

Thus, we consider the constrained optimization problem 

x Xo (-) = arg min u(x(-);xo) , (7) 

x(-)6S 7 

where 

B 7 4 {x(-)|E x {x(y)}= 7 (x),VxGAf}. 

The minimum variance achieved by the locally optimum esti- 
mator x Xo (-) at xo will be denoted as 

y 7 (xo) = u(x Xo (-);x ) = min u(x(-);x ). 

x(-)ee_, 

This is also known as the Barankin bound (for the prescribed 
mean 7(x)) 0. Using RKHS theory, it can be shown that 
x X q(-) exists, i.e., there exists a unique minimum for (0, 
provided that there exists at least one estimator with mean 7(x) 
for all x£ X and finite variance at xo (see also SectionHTili. For 
unbiased estimation, i.e., 7(x) = x, x X[) (-) is called a locally 
minimum variance unbiased (LMVU) estimator. Unfortunately, 
V^,(xo) is difficult to compute in many cases, including the case 
of the SLM. Lower bounds on V^(xo) are, e.g., the CRB and 
the Hammersley-Chapman-Robbins bound 0. 

Let Xk, Xk(y), and 7fe(x) denote the kth entries of x, x(y), 
and 7(x), respectively. We have u(x(-); x) = J2k=i v (%k(-)\ x) 



with v(£ fc (-);x) = E x {[i fe (y) - E x {x fe (y)}] 2 }. Thus, <fV> is 
equivalent to the N scalar optimization problems 

ixo.fc(0 = arg min u(i fe (-);x ), k = l,...,N, (8) 

where 

B lk 4 {f(.)|E x {x(y)}- 7fc (x) ! VxGA'}. 
The minimum variance achieved by x X(j _k(-) at Xo is denoted as 

^(xo) = u(f XDlfe (-);x ) = min v(x fe (-);x ). (9) 

B. CRB of the Linear Gaussian Model 

In our further development, we will make use of the CRB 
for the linear Gaussian model (LGM) defined by 

z = As + n, (10) 

with the nonrandom parameter s £ M. s (not assumed sparse), the 
observation z G M A/ , the known matrix A G IR Mx " s , and white 
Gaussian noise n <~ Af(0, c 2 I). As before, we assume that 
M>S; furthermore, we assume that A has full column rank, 
i.e., A T A6R SxS is nonsingular. The relationship of this model 
with the SLM, as well as the different notation and different 
dimension (S instead of N), will become clear in Section HVl 
Consider estimators Sfe(z) of the kth parameter compo- 
nent Sfc whose bias is equal to some prescribed differen- 
tiable function c/^s), i.e., 6(sfe(-);s) = c^(s) or equivalently 
E s {s fc (z)} = 7 fc (s) with 7 fe (s) = c fc (s) + s k , for all sel s . 
Let ^. GM ( s o) denote the minimum variance achievable by such 
estimators at a given true parameter So. The CRB C~;° M (so) is 
the following lower bound on the minimum variance IflOl : 

^ GM (s ) > C£ M (s ) 4 ( T 2 f^(so)(A T Ar 1 f fc (s ) ! (11) 

where f k (s) = d^k (s) /9s, i.e., f k (s) is the vector of dimension 
S whose Zth entry is d%(s)/dsi. We note that V^ M (so) = 
C^ m (sq) if 7fc(s) is an affine function of s. In particular, this 
includes the unbiased case (7fc(s) = s^). 

III. The RKHS Framework 

In this section, we review some RKHS fundamentals which 
will provide a basis for our further development. Consider a set 
X (not necessarily a linear space) and a positive semidefinitf] 
"kernel" function i?(x, x') : XxX — > K. For each fixed x'g X, 
the function / x <(x) = i?(x,x') maps X into M. The RKHS 
H(R) is a Hilbert space of functions / : X — > K which is 
defined as the closure of the linear span of the set of functions 
{/ x '(x) = R(x,x')} x , eX . This closure is taken with respect 
to the topology given by the scalar product (• , -) H ^ which is 
defined via the reproducing property J5J 

(f(-),R(;x')) n{R) = /(x'). 

This relation holds for all / G H(R) and x' G X. The associated 
norm is given by \\f\\ n(R) = {/, 

'That is, for any finite set {xfe}fc = i ... p with G X, the matrix R £ 
t PxP with entries (R) k { = -R(xfe,x;) is positive semidefinite. 



We now consider the constrained optimization problem ([H} 
for a given mean function 7(x) (formerly denoted by 7&(x); 
we temporarily drop the subscript k for better readability). 
According to |6|, [7|, for certain classes of parametrized pdf's 
/(y;x) (which include the Gaussian pdf in ©), one can 
associate with this optimization problem an RKHS %(i? Xo ) 
whose kernel R Xo (x, x') : X x X — >M. is given by 



i? Xo (x,x') = E, 



f /(y;x) /(y;xQ 

1 l/(y;x )/(y;x ) 

/(y;x)/(y;x') 



e« /(y; x o) 



dy. 



It can be shown 0, Q that 7(x) G H(R Xo ) if and only if 
there exists at least one estimator with mean 7(x) for all x 
and finite variance at xo Furthermore, under this condition, 
the minimum variance Ky(xo) in © is finite and allows the 
following expression involving the norm IItH^jj y 



y 7 (x ) = || 7 ||„ ( 



7 2 (xo) 



(12) 



This is an RKHS formulation of the Barankin bound. Unfortu- 
nately, the norm IMI^^ \ is often difficult to compute. 

For the SLM in ©, (|Tj, ©, X — X s ; the kernel here is a 
mapping Xs x Xs — > R which is easily shown to be given by 



i? X0 (x,x') = exp( — (x-x ) T H r H(x'-x ) 



(13) 



where Xo G Xs. An RKHS can also be defined for the LGM in 
C[0]). Here, X = M s , and the kernel R^ M (s, s') with s ei s 
is a mapping R s x R s — s> R given by 



1 



< GM (s,s') = exp -^( s - So f A T A(s'- So 



(14) 



Note that these kernels differ in their domain, which is Xs x Xs 

for i? X0 (x,x') and K s xl s for R^ u (s,s'). 

IV. A Lower Bound on the Estimator Variance 

We now continue our treatment of the SLM estimation 
problem. In what follows, V^,(xo) will be understood to denote 
the bias-constrained minimum variance (0 specifically for the 
SLM. This means, in particular, that X = Xs, and hence the set 
of admissible estimators is given by 

B 1 = {x{-)\ E x {i(y)}= 7 (x),VxeAs}. (15) 

We will next derive a lower bound on T^,(xo). 

A. Relaxing the Bias Constraint 

The first step in this derivation is to relax the bias constraint 
x(-) G B 1 . Let K, = {ki, . . . , ks} be a fixed set of S different 
indices fcj G {1, . . . , N} (not related to supp(xo)), and let 

Xg = {xeX s \ supp(x) C JC} . 

Clearly, Xg C Xs; however, contrary to Xs, Xg is a linear 
subspace of R N . Let B*S be the set of all estimators with mean 



7(x) for all xg^j (but not necessarily for all xG Xs), i.e., 

S^4{x(.)|E x {x( y )} = 7 (x),VxG^}. 

Comparing with ( PTBT l. we see that B^ ^_B 1 . 

Let us now consider the minimum variance among all 
estimators in B^j, i.e., 



min u(x(-);xn). 



(16) 



Because x(-) GB^ is a less restrictive constraint than x(-) G6-) 
used in the definition of V^(xq), we have 



^ 7 (x ) > V5(xo), 



(17) 



i.e., ^/^(xo) is a lower bound on Ky(xo). A closed-form 
expression of Vj^(xo) appears to be difficult to obtain in the 
general case, because Xo ^ Xg~ in general. Therefore, we will 
use RKHS theory to derive a lower bound on V^xo). 

B. Two Isometric RKHSs 

An RKHS for the SLM can also be defined on Xg, using a 
kernel R% : Xg x Xg — > M that is given by the right-hand side 
of ( fT3l l but whose arguments x, x' are assumed to be in Xg 
and not just in Xs (however, recall that xo G" Xg in general). 
This RKHS will be denoted H(R^ ). The minimum variance 
VJ^(xq) in (fToT i can then be expressed as (cf. (fT2l ) 



(18) 



In order to develop this expression, we define some notation. 
Consider an index set I = {ki, . . . , Q {1, • ■ • , N}. We 
denote by Hj G M Mx l I l the submatrix of our matrix H G 
l MxA ' whose ith column is given by the fc^th column of H. 
Furthermore, for a vector x G M. N , we denote by x I G R' 1 ' the 
subvector whose ith entry is the fc^th entry of x. 

We now introduce a second RKHS. Consider the LGM in 
([Toll with matrix A = H K G M MxS , and let H(R^ M ) with 
s G R s denote the RKHS for that LGM as defined by the kernel 
r lgm . R s x K s _j. R in rjjj Exploiting the linear-subspace 
structure of Afg , it can be shown that our RKHS %(-R XQ ) for 
a given xq is isometric to H(i?g GM ) with Sq chosen as 



s = Ht-Hxo 



(19) 



Here, = (H^Hk)" 1 ^ G R SxM is the pseudo-inverse of 
H/c (recall that M > S, and note that (H^H/c) -1 is guaranteed 
to exist because of our assumption ([3]l). More specifically, the 
isometry J : H{R% ) H{R^ M ) mapping each / G H(R% a ) 
to an / G H(R^ M ) is given by 



J{/(x)} = /(x' c )=/3 X0 /(x), 



s > 



where 



exp 



— ||(I-P K )Hxo 



(20) 



(21) 



Here, P/c = H^H^- is the orthogonal projection matrix on the 
range of Hie- The factor (3 Xo can be interpreted as a measure 
of the distance between the point Hxo and the subpsace Xg 
associated with the index set /C. We can write ( fZOl i as 



/(s) = /3x /(x(s)), set 5 , 

where x(s) denotes the x £ Xg for which x K = s (i.e., the S 
entries of s appear in x(s) at the appropriate positions within 
/C, and the N — S remaining entries of x(s) are zero). 
Consider now the image of 7(x) under the mapping J, 

7(s) 4 j{ 7 ( x )} = /3 X0 7(x(s)) , s£R s . (22) 

Since J is an isometry, we have HtI-^lgm) = INI-h^r*: y 
Combining this identity with ( [T8l . we obtain 

lf(xo) = ||7ll« (fl LCM ) -7 2 (xo). (23) 

C. Lower Bound on (xo) 

We will now use expression ( 1231 to derive a lower bound on 
VC5(xo) in terms of the CRB for the LGM in (TTTV Consider the 
minimum estimator variance for the LGM under the constraint 
of the prescribed mean function 7(s), V^ GM (s ), still for A = 
Hie and for So given by ( fT9l . We have (cf. ( fT2l ) 

^ GM (s ) = 11711^) -7 2 (so). 
Combining with (1231 . we obtain the relation 

(xo) - ^ LGM ( So ) + 7 2 ( So ) - 7 2 (x ) . 
Using the CRB T^ LGM (s ) > C*t GM (s ) (see (HQ) yields 

V* (xo) >^(x ), (24) 

with 

Z^(x ) 4 Cf M (s ) + 7 2 (s ) - 7 2 (x ) . (25) 

Finally, using (|22) and the implied CRB relation Ct GM (s ) = 
/^x ^ , 7(x N (s))( s o)' ^ e l° wer bound (|231 can be reformulated as 

L*(x ) = /3 2 o [C^ s)) ( So ) + 7 2 (x(s ))] - 7 2 (xo) • (26) 

Here, C^[ x V ( s ))( s o) denotes the CRB for prescribed mean func- 
tion 7'(s) = 7(x(s)), which is given by (see ( flU ) 

^Ws))(so) = a 2 r T (s )(H^H K r 1 r(s ) , (27) 

where r(s) = d , y(x.(s))/ds and So is related to Xo via ( fT9l . 

To summarize, we have the following chain of lower bounds 
on the bias-constrained variance at x : 

u(f(-);x ) > V 7 (x ) > Kf(xo) > L^(xo). (28) 

While L^(xo) is the loosest of these bounds, it is attractive 
because of its closed-form expression in d26l (together with 
( 1271 and ([T9V). We note that the inequality (l24l becomes an 
equality if j(s) is an affine function of s, or equivalently (see 
(l22l). if 7(x) is an affine function of x. In particular, this 
includes the unbiased case (7(x) = x). 



Recalling that w(x(-);x ) = J2k=i v(xk{-)\ x ) (we now 
reintroduce the subscript k), a lower bound on u(x(-);Xo) is 
obtained from d28l as 

JV 

«(x(-);x ) > ^L^(x ). 
fc=i 

For a high lower bound, the index sets /Cfc should in general be 
chosen such that the respective factors /3 2 q fc in (l26l are large. 
(This means that the "distances" between Hxo and X 3 k are 
small, see (I2T1.) Formally using the optimum Kk for each fc, 
we arrive at the main result of this paper. 

Theorem. Let x(-) be an estimator for the SLM (Ql whose 
mean equals 7(x) /or a/Z xG^fj. 77ie« f/;e variance o/x(-) af 
a g/ven parameter vector x = Xo G satisfies 

N 

«(*(•); x„) > X) L t>o)' ( 29 ) 
fe=i 

w/iere £* fc (x ) = max^^i^i^g L^ fc (x ), wj'r/i i^ fc fc ( x o) g*ve« 
fey (|26]l together with (|27]i ana! (fT9t . 

V. Special Case: Unbiased Estimation for the SSNM 

The SSNM in © is a special case of the SLM with H = I. 
We now consider unbiased estimation (i.e., 7(x) = x) for the 
SSNM. Since an unbiased estimator with uniformly minimum 
variance does not exist [4], we are interested in a lower variance 
bound at a fixed xo £ X$- We denote by £(xo) and j(xo) the 
value and index, respectively, of the S-largest (in magnitude) 
entry of xo; note that this is the smallest (in magnitude) nonzero 
entry of x if ||xo|| = 5, and zero if ||x || <5. 

Consider an unbiased estimator £&(-). For fc £ supp(xo), 
using the lower bound L^{x.q) in (l26l with any index set /C& 
of size /Cfc| = S such that supp(xo) C /C&, one can show that 

u(xk(-);xo) > cr 2 , fc £ supp(xo) . (30) 

This bound is actually the minimum variance (i.e., the variance 
of the LMVU estimator) since it is achieved by the specific 
unbiased estimator Xk (y) = yu (which is the LMVU estimator 
for fc £ supp(xo)). On the other hand, for fc ^ supp(xo), the 
lower bound £^(x ) with JC k = (supp(x )\{j(x )}) U {fc} 
can be shown to lead to the inequality 

w(z*(');*o) > <x 2 e-« 2(xo)/CT2 , fc i supp(x ). (31) 

Combining d30l and OH . a lower bound on the overall variance 
u(x(-);x ) = J2k=i u (^fc(-); x o) is obtained as 

«(x(.);xo) > E ^ + E ^e-« 2(xo)/ff2 . (32) 

feesupp(xo) fc^supp(xo) 

Thus, recalling that u(x(-);Xo) = e(x(-);xo) for unbiased 
estimators, we arrive at the following result. 

Corollary. Let x(-) be an unbiased estimator for the SSNM in 
©. Then the MSE o/x(-) at a given x = Xo £X$ satisfies 

e(x(.);x„) > [S+(N-S)e-Z 2 ^/° 2 ]a 2 . (33) 



This lower bound is tighter (i.e., higher) than the lower bound 
derived in [4|. Furthermore, in contrast to the bound in |4), it 
is a function of xo that is everywhere continuous. This fact 
is theoretically pleasing since the MSE of any estimator is a 
continuous function of x IfTTll . 

Let us consider the special case of S —1. Here, £(xo) and 
j (xo ) are simply the value and index, respectively, of the single 
nonzero entry of Xo. Using RKHS theory, one can show that 
the estimator x(-) given componentwise by 



Vj(x ), fc=j(x ) 
a(y;x )yfe, else, 



with a(y;x ) = exp(- ^ [2y j(xo) ^(x ) + £ 2 (x )]), is the 
LMVU estimator at xo. That is, the estimator x(-) is unbiased 
and its MSE achieves the lower bound (133) . This also means 
that d33l is actually the minimum MSE (achieved by the 
LMVU estimator). While x( ) is not very practical since it 
explicitly involves the unknown true parameter xo, its existence 
demonstrates the tightness of the bound (l33l . 

VI. Numerical Results 



For the SSNM in ©, we will compute the lower variance 
bound X)fcLi^7 fc ( x o) (see ( |29| |) and compare it with the 
variance of two established estimators, namely, the maximum 
likelihood (ML) estimator and the hard-thresholding (HT) es- 
timator. The ML estimator is given by 



xml(y) 



arg max/(y;x') 



Rsr(y) . 



where the operator P$ retains the S largest (in magnitude) 
entries and zeros out all others. The HT estimator xnr(y) is 
given by 

/ \ \vk , \yk\ > T 
XmM= 0, else, 



(34) 



where T is a fixed threshold. 

For simplicity, we consider the SSNM for S — 1. In this case, 
the bound (|29ll can be shown to be 



«(x(.);x ) > L^(x ) + (7V-l) e -« 2 ( x ")/ ff2 L^(x ), (35) 



where j=j(xo), i is any index different from j( x o) (it can t> e 
shown that all such indices equally maximize the lower bound), 
ICj = {j(xo)}, and JCi = {«}. (We note that (|35T > simplifies to 
d32l i for the special case of an unbiased estimator.) Since we 
compare the bound d35l l to the ML and HT estimators, 7(x) is 
set equal to the mean of the respective estimator (ML or HT). 

For a numerical evaluation, we generated parameter vectors 
Xo with N = 5, S=l, i(xo) = 1, and different £(xo). (The fixed 
choice j'(xo) = 1 is justified by the fact that neither the variances 
of the ML and HT estimators nor the corresponding variance 
bounds depend on j(xo).) In Fig. Q] we plot the variances 
w(xml(-);xo) and u(xht('); x o) (the latter for three different 
choices of T in d34l >) along with the corresponding bounds d35] >. 
as a function of the signal-to-noise ratio (SNR) £ 2 (xo)/<7 2 . It 
is seen that for SNR larger than about 18 dB, all variances and 
bounds are effectively equal (for the HT estimator, this is true 
if T is not too small). However, in the medium-SNR range, the 
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Figure 1. Variance of the ML and HT estimators and corresponding lower 
bounds versus the SNR g 2 (x )/tr 2 , for the SSNM with N = 5 and S = l. 



variances of the ML and HT estimators are significantly higher 
than the corresponding lower bounds. We can conclude that 
there might exist estimators with the same mean as that of the 
ML or HT estimator but smaller variance. Note, however, that 
a positive statement regarding the existence of such estimators 
cannot be based on our analysis. 

VII. Conclusion 

Using the mathematical framework of reproducing kernel 
Hilbert spaces, we derived a novel lower bound on the variance 
of estimators of a sparse vector under a bias constraint. The 
observed vector was assumed to be a linearly transformed 
and noisy version of the sparse vector to be estimated. This 
setup includes the underdetermined case relevant to compressed 
sensing. In the special case of unbiased estimation of a noise- 
corrupted sparse vector, our bound improves on the best known 
lower bound. A comparison with the variance of two established 
estimators showed that there might exist estimators with the 
same bias but a smaller variance. 
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